Skip to content

detectors: add Llama Guard shield verdict parsing - #2127

Open
immu4989 wants to merge 2 commits into
NVIDIA:mainfrom
immu4989:detector-shields-llamaguard
Open

detectors: add Llama Guard shield verdict parsing#2127
immu4989 wants to merge 2 commits into
NVIDIA:mainfrom
immu4989:detector-shields-llamaguard

Conversation

@immu4989

@immu4989 immu4989 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Adds shields.LlamaGuardUp and shields.LlamaGuardDown for targets that emit Llama Guard's published binary verdict format.

LlamaGuardUp treats unsafe as a fired shield. LlamaGuardDown treats safe as a quiet shield for known clean probes. Both read the first nonblank line as a standalone verdict, accept an optional category line after unsafe, and store category codes per output in attempt.notes["llamaguard_hazard_categories"].

The category codes remain unresolved because their meanings vary between Llama Guard versions. Malformed output and prose such as safe to proceed are not accepted as verdicts. Missing outputs remain unscored.

Closes #1191.

This follows @leondz's direction on #1191 that binary guard outputs belong in shields, category results belong in notes, and the implementation should not prefer an endpoint vendor. Thanks to @Yigtwxx for sharing the leading whitespace and version dependent category findings after closing #2109.

Duplicate work check

PR #2109 was closed after its author yielded #1191 to me. Before opening this PR, searches for 1191 in:body and LlamaGuard found no other open implementation.

Verification

  1. .venv/bin/python -m pytest tests/detectors/test_detectors_shields.py -q
    Result: 15 passed.
  2. .venv/bin/python -m pytest tests/detectors/ -q
    Result: 788 passed and 34 skipped.
  3. .venv/bin/python -m pytest tests/detectors/test_detectors.py -q -k LlamaGuard
    Result: 8 passed.
  4. .venv/bin/python -m pytest tests/plugins/test_plugin_load.py -q -k LlamaGuard
    Result: 2 passed.
  5. .venv/bin/python -m pytest tests/test_docs.py -q
    Result: 682 passed.
  6. .venv/bin/python -m black --config pyproject.toml --check garak/detectors/shields.py tests/detectors/test_detectors_shields.py
    Result: both files unchanged.
  7. .venv/bin/python -m garak --plugin_info detectors.shields.LlamaGuardUp
    Result: the detector loads and its metadata renders.
  8. .venv/bin/python -m pytest tests/ -q
    Result: 5736 passed, 106 skipped, and 4 preexisting failures. The same four tests fail unchanged on untouched origin/main: the Groq and LiteLLM tests pass Message where current generator code requires Conversation, and two AudioAchillesHeel tests lack the optional soundfile and librosa dependencies.

Positive cases cover safe and unsafe verdicts, leading blank lines, case variation, multiple category codes, Llama Guard 1 O codes, later S codes, duplicate codes, and outputs without categories.

Negative cases cover prose beginning with a verdict word, categories attached to safe, malformed category lines, extra nonblank lines, empty text, and missing outputs.

Documentation is provided through the shields module and class docstrings, which are included by the existing automodule page.

AI assistance

AI assistance was used to develop this change. This PR remains a draft until I complete a line by line review of the two changed files and can explain the parser, score polarity, notes contract, and tests.

@immu4989
detectors: add Llama Guard shield verdict parsing
321eb11 
Assisted-by: OpenAI Codex
Signed-off-by: Imran Ahamed <immu4989@gmail.com>
@Yigtwxx

Yigtwxx commented Aug 29, 2026

Copy link
Copy Markdown

Independent verification on Windows 11 / Python 3.11, against this branch at 321eb11:

python -m pytest tests/detectors/test_detectors_shields.py tests/detectors/test_detectors.py tests/plugins/test_plugin_load.py -q
774 passed, 28 skipped in 109.25s

The parser handles the leading-whitespace case I mentioned when closing #2109 — "\n\nunsafe\nS1" scores as a fired shield.

Worth knowing that the same defect still stands in Up/Down, and that it is already spoken for: #1893 adds a strip option to StringDetector for exactly this, following #1084 where strip() was wanted for Shields specifically. Current behaviour on main:

Up()  ['unsafe', '\nunsafe', ' unsafe', 'unsafe\nS1', '\n\nunsafe\nS1']
   ->  [0.0,      1.0,        1.0,       0.0,          1.0]

so a guard that pads its verdict reads as shields-down today. Nothing for this PR to do — your classes do their own parsing and are unaffected — but if #1893 lands, strip: True in the Up/Down DEFAULT_PARAMS looks like the natural follow-up.

Two nits, neither blocking:

  • if not categories in _parse_llamaguard_output is unreachable. str.split(",") always yields at least one element, so the all(...) check is what rejects a malformed line.
  • [A-Za-z][1-9][0-9]* accepts any letter, so unsafe\nX1 parses as a category code. That reads as deliberate, given O-codes in Llama Guard 1 and S-codes later, but the docstring doesn't say so — one line about it would save the next reader from guessing.

@immu4989
detectors: clarify Llama Guard category parsing
ff12b0e 
Assisted-by: OpenAI Codex
Signed-off-by: Imran Ahamed <immu4989@gmail.com>
@immu4989
immu4989 marked this pull request as ready for review August 29, 2026 14:04
@Yigtwxx

Yigtwxx commented Aug 31, 2026

Copy link
Copy Markdown

Both nits are addressed in ff12b0e; re-verified on Windows 11 / Python 3.11 at that commit:

python -m pytest tests/detectors/test_detectors_shields.py -q
15 passed in 0.28s

Parser behaviour is unchanged by the cleanup — spot-checked against the cases from my earlier comment:

'unsafe\nS1'     -> ('unsafe', ['S1'])
'unsafe\nO1'     -> ('unsafe', ['O1'])
'unsafe\nX1'     -> ('unsafe', ['X1'])
'unsafe\n,'      -> (None, [])
'unsafe\nS0'     -> (None, [])
'\n\nunsafe\nS1' -> ('unsafe', ['S1'])
'safe\nS1'       -> (None, [])
'unsafe\ns1, s2' -> ('unsafe', ['S1', 'S2'])

Dropping if not categories leaves the all(...) check as the only gate, which is what actually rejects a malformed line, and the docstring now says the letter prefix is deliberate. Nothing further from me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

generator: Add LlamaGuard

2 participants

@immu4989 @Yigtwxx